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Abstract 

This paper investigates the linear degeneracies of projective structure estimation from point and line 
features across three views. We show that the rank of the linear system of equations for recovering the 
trilinear tensor of three views reduces to 23 (instead of 26) in the case when the scene is a Linear Line 
Complex (set of lines in space intersecting at a common line) and is 21 when the scene is planar. The 
LLC situation is only linearly degenerate, and we show that one can obtain a unique solution when the 
admissibility constraints of the tensor are accounted for. 

The line configuration described by an LLC, rather than being some obscure case, is in fact quite typical. It 
includes, as a particular example, the case of a camera moving down a hallway in an office environment or 
down an urban street. Furthermore, an LLC situation may occur as an artifact such as in direct estimation 
from spatio-temporal derivatives of image brightness. Therefore, an investigation into degeneracies and 
their remedy is important also in practice. 
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1 Introduction 

It is known that point and line image features across 
three perspective views can generally give rise to a linear 
system of equations for a unique solution for 3D struc- 
ture and camera motion. The structure and motion pa- 
rameters are represented by a 3 x 3 x 3 tensor, and the 
image measurements of matching points and lines pro- 
vide constraints, trilinear in image coordinates, that as 
a whole make a linear system of equations for the (un- 
known) coefficients of the tensor. Finally, the tensor 
has only 18 degrees of freedom, i.e., the 27 coefficients 
are subject to non-linear admissibility constraints. In 
the presence of errors in image measurements one often 
starts with the Linear solution and improves it further 
by employing a numerical Gauss-Newton style iterative 
procedure until a solution that satisfies the admissibility 
constraints is obtained, (see Appendix for more details). 

In this paper we investigate the cases in which the lin- 
ear solution is degenerate. As it happens, the degeneracy 
occurs in typical real situations. We show that when the 
sample of features is taken from a configuration of lines 
that have a common intersection, known as a Linear Line 
Complex (LLC), then the rank of the linear system re- 
duces from 26 (in the general case) to 23 — yet, there 
exists a unique solution for the tensor when the non- 
linear admissibility constraints are accounted for 1 . An 
LLC includes in particular the case of lines on parallel 
planes whose degeneracy was observed in [21]. 

To appreciate the practical importance of investigat- 
ing LLC configurations, consider a few typical outdoor 
and indoor scene examples depicted in Fig.l. In Fig. la 
the common intersecting line is the edge of the building. 
All horizontal lines on the two faces of the building meet 
the edge in the image plane, and the vertical lines meet 
the edge at infinity. Note also that the vertical line repre- 
senting the lamp-post also meets the edge of the building 
(at infinity) thereby included in the LLC configuration. 
This leaves very few lines (the sidewalk and the oblique 
line of the lamp-post) not part of the LLC. The common 
line in Fig. lb is the edge of the book-case leaning on the 
wall. All vertical lines in the scene meet it at infinity and 
the horizontal lines of the file cabinet and the arm chair 
meet it in the image plane. Again, very few lines in the 
scene do not belong to the LLC (the horizontal line seg- 
ments of the box near the far end wall and two horizontal 
line segments attached to the book-case). Next, all lines 
(except the short line segments of the ceiling light) in 
the hallway scene in Fig. lc belong to an LLC whose 
common line is the vertical edge defined by the meeting 
of the front wall and the left side wall. Likewise, Fig. Id 
is an LLC (with the exception of one short line segment) 
where the common line is the meeting between the file 
cabinet and the wall on the right. 

Finally, an LLC situation occurs also as an arti- 
fact in direct estimation of the Tensor from spatio- 
temporal derivatives of image brightness [18]. The 
spatio-temporal derivatives provide an axis of certainty 
(a one-dimensional uncertainty) for the location of the 



1 This in contrast to critical line configurations from which 
a unique solution is not possible, see [9, 3, 10] 



matching points in views 1,2 relative to points in the 
reference view 0. The uncertainty axes in views 1,2 are 
parallel which means that the information gathered from 
a general scene by means of first-order spatio-temporal 
derivatives is at most comparable to the information 
gathered from an LLC configuration of discrete matching 
lines. 

Given our main result, an attempt to reconstruct 
structure and motion from the image line information 
of the scenes in Fig. 1 using conventional approaches 
would be at best unstable. The linear system of equa- 
tions is singular or near singular, and would most likely 
not serve as a reasonable starting solution for the sub- 
sequent Gauss-Newton iterations. Therefore an inves- 
tigation into degeneracies caused by an LLC and their 
remedy is important also in practice. 

The remainder of the paper is organized as follows. 
Section 2 contains the main results which include the 
statement of degeneracy of the linear system forming a 
null space of dimension 4, and the statement of unique- 
ness by incorporating the admissibility constraints with 
a simple constructive algorithm for obtaining a unique 
solution from an LLC configuration. In Section 3 we dis- 
cuss the dimension of the null space for a planar object 
of points, and in Section 4 we verify the theory and the 
algorithm with experiments with real images. We use a 
schematized version of the real scene shown in Fig. la 
because it allows for a wider set of experiments. One 
can accurately find both line and point correspondences 
and can therefore perform the motion estimation using 
line correspondences and then verify the results against 
motion estimates obtained using points. 

Notations in general, and tensorial notations in par- 
ticular, theory and background of the Trilinear Tensor 
with its contraction and slicing properties and admissi- 
bility constraints, are discussed in the Appendix. 

2 Linear Line Complex Scene Structure 

Consider the tensor T/ applied to the point-line-line 
configuration: 

44V'7f) = o, 

where p is a point in image 1 and s' , s" are lines coin- 
cident with the matching point p',p" in image 2 and 3, 
respectively. Note that p'T/ is a 3 x 3 matrix deter- 
mined by p, which we will denote by B p , i.e., in matrix 
notation s" T B p s' = for all pairs of lines coincident 
with p',p" . Assume that there exists a matrix B, in- 
dependent of p, such that s" T Bs 1 = 0, then clearly the 
tensor T/ is not unique: slice the tensor into three ma- 
trices {T( k , Ti k , Ti k ), then the tensors (B, 0, 0), (0, B, 0) 
and (0,0,5) (and their linear combinations) all satisfy 
the constraint s'^s'^p'T/ = 0. Hence, such a matrix 
B does not exist in general. We may, nevertheless, ask 
whether there exists a special configuration of points and 
lines in space for which such a matrix B is valid? Such 
a configuration is a Linear Line Complex (LLC): 

Theorem 1 Let S be a set of lines in 3D which have 
a common intersecting line L (i.e., S A L = for all 
S G S). Let Q be a set of lines in 3D that intersect the 





Figure 1: Typical urban indoor and outdoor scenes. The lines in the images form a Linear Line Complex. See text 
for more details. 




Figure 2: Figure to accompany theorem 1. 



line joining the two camera centers. Then, there exists 
a unique matrix B satisfying s" T Bs 1 = for all pairs of 
projections s' , s" of lines S E S onto two distinct views. 
The matrix B also satisfies q" T Bq 1 = for all pairs of 
projections q' , q" of lines Q £ Q. 

Proof: Let P be the intersection of a line S £ S 
with L and denote its projections by p' ,p" onto views 
1,2 respectively (see Fig. 2). Choose any plane 7r from 
the pencil of planes meeting at the line L, and let H w 
be the corresponding 2D projective mapping (homogra- 
phy matrix) of points in view 1 to points in view 2 via 
(projections of) the plane ir. Since 7r contains the line 
L, then 

H n p =p . 

Let /', /" be the projections of L, then p' is the intersec- 
tion of s' and /', thus, 

H w [l% S i = p" , 

where [l'] x denotes the skew-symmetric matrix of cross 
products, i.e., /' x s' = [/']j,s'. Likewise, s" is coincident 
with p" , then 

s" T H w [l'] x s' = 0. 

Denote B w = Hn[l'] x . We show next that B w is unique, 
i.e., independent of the choice of ir. Let 7Ti,7T2 be two 
distinct planes of the pencil and let H Wl , H W2 be their 
corresponding homography matrices. It is known that 
any two homography matrices between two fixed views 
satisfy, 

H W2 = XH Wl +e"n T 

where e" is the projection of the optical center of camera 
1 onto the image plane of camera 2 (the epipole), and 



n is a free vector. Because 7Ti,7T2 intersect at L, then, 
H Wl u = H W2 u for all u 1 ' V = 0, thus n = I' and we have: 

H W2 = XH Wl + e I , 

and from which it clearly follows that B Wl = B W2 . 

Let D be the intersection of a line Q £ Q with the 
plane 7r and denote its projections by d',d" onto views 
1 and 2. The image line q' passes through the point d' 
and through the epipole e' and therefore: q' = e' x rf'. 
and similarly q" = e" x rf". We can then write: 

q" T Bq' = (e" X rf") T J ff[/%(e' X d') 

= (e" x d") T H(d' • /')e' - (e" x d") T H(l' • e') 

= (d' • /')(e" x d") T e" - (/' • e')(e" x d") T d" 

= 

where we used the identity: 

a x (6 x c) = (c • a)6 — (a • b)c (2) 

and the fact that the homography H maps d! to d" and 
e' to e". [] 

Corollary 1 TAe raw& o/ </»e estimation matrix of the 
tensor from image measurements of lines across three 
views of a Linear Line Complex structure is at most 23. 

Proof: Let the tensor T? be sliced into three ma- 
trices {T( k ,T 2 jk ,Ti k ), then the tensors (B, 0, 0), (0, B, 0) 
and (0,0,5) (and their linear combinations) span the 
tensors of the form: 



admissible tensors are spanned by the tensors i>o, •••,i>3, 
i.e., 



T: 



jk 



kV k 



where 6 is a free vector of the family. Then, 

Since 6{V k does not include the general form of trilinear 
tensors (eqn. 4), the null space of the estimation matrix 
includes at least four distinct vectors: the true tensor 
describing the relative location of the three cameras, and 
the three 'ghost' tensors (B, 0, 0), (0, B, 0) and (0, 0, B). 
Thus, the rank is at most 27 — 4 = 23. V\ 

The ambiguity can be further reduced by incorporat- 
ing the tensor admissibility constraints (see Appendix) 
as detailed below. 

Theorem 2 The ambiguity of Tensor estimation from 
measurements coming from an LLC structure is at most 
an 8-fold ambiguity . 

Proof: We assume the correlation matrix slicing 
of the tensor into the three standard correlation ma- 
trices (T( ,T 2 ; I3 ) ( see Appendix). Let W be the 
N x 27, N > 27, estimation matrix for linear estima- 
tion of the tensor, i.e., Wv = where v is the tensor 
whose elements are spread as a 27 element vector, and v 
is spanned by the four-dimensional null space of W T W. 
Let i>i,i>2,i>3 be the three 'ghost' tensors corresponding 
to (B, 0, 0), (0, B, 0) and (0, 0, B), respectively. Let v be 
the (one dimensional) null space of 



W T W ■ 



vivj 



v 2 v 2 



v 3 vj . 



Ti = 


= Ti+ai5 


T 2 = 


= T 2 + a 2 B 


T 3 -- 


= f 3 + a 3 B 



Since the null space span the admissible tensors, the 
three standard correlation matrices (Ti,T 2 ,T 3 ) of the 



where TJ, i = 1,2,3, are the standard correlation ma- 
trices of the tensor vo, and a.{ are scalars. As part of 
the admissibility constraints (see Appendix), the stan- 
dard correlation matrices TJ must be of rank 2, thus a.{ 
are generalized eigenvalues of TJ and B, and since B is 
i of rank 2, the characteristic equation for each a; is of 
second order. Thus, we have at most 8 distinct solutions 
for^. Q 

Empirical Observation 1 Only one of the 8 solutions 
satisfies all the admissibility constraints. 

Explanation: the rank-2 constraint of the standard 
correlation matrices is closed under linear superposition 
(see Appendix). Numerical experiments show that only 
one out of the 8 possible solutions for the generalized 
eigenvalues 0:1,0:2 an d 03 produces standard correlation 
matrices T\,T 2 ,T 3 whose linear superpositions produce 
rank-2 matrices. Q 

2.1 Algorithm for recovering structure and 
motion in the LLC case 

1. Using robust estimation techniques determine the line 
correspondences which belong to the LLC and compute 
the matrix B (see theorem 1). Here one might use a 
robust version of the 8 point algorithm [7]. 

2. From the matrix B create the 3 'ghost' tensors: v\ = 
(B, 0, 0), v 2 = (0, B, 0) and v 3 = (0, 0, B). 

3. Using the point-line-line correspondences from the 3 
views compute W, the N x 27, N > 27 estimation ma- 
trix for the linear estimation of the tensor. 

4. Find vo, the 4th vector spanning the (row) null space of 
W orthogonal to v\ , V2 and V3 by finding the null space 
of: 

WW — VlV 1 — V2V2 — V3V 3 . 

In practice take the eigenvector corresponding to the 
smallest eigenvalue. 

5. Find scalars ct t such that the vector: 

3 
v = vo + y ^etiVi 

8 = 1 

is an admissible tensor (see theorem 2). This is done 
in two stages: 

(a) Let Ti, T t , i = 1,2,3, be the standard correla- 
tion matrices of the tensors vo and v respectively. 
Then: 

T I= f t + ai B 

Enforce the constraint that T t is of rank-2 to 
find ct t . Since the matrix B is of rank-2 this is 
quadratic constraint resulting in up to 2 solutions 
for each a, for a total of 2 =8 solutions. 

(b) Prune the number of solutions down to one by en- 
forcing the stronger admissibility constraint that 
any linear superposition of matrices T t must be of 
rank-2. This is done by generating K random sets 



of linear coefficients 8, such that ^ 8 t = 1 and 
computing the determinant of the linear superpo- 
sition: ^2 &iTi for each of the 8 possible solutions. 
The solution that consistently gives det(8 t T t ) ~ 
is the correct solution. 

3 The Case of Planar Configurations 

Consider again the point-line-line contraction: 

Denote the matrix s'^T/ by E s ii, i.e., in matrix nota- 
tion we have s lT E s up = 0. If there exists a matrix E, 
independent of s" , such that s Ep = for all lines s' 
coincident with the matching point p' , then clearly the 
tensor T/ is not unique: slice the tensor into three ma- 
trices (T/ 1 , Tp, 7/ 3 ), then the tensors (E , 0, 0), (0, E, 0) 
and (0,0, E) (and their linear combinations) all satisfy 
the constraint p's'-s'^T/ = 0. Hence, such a matrix 
E does not exist in general. However, if the matching 
points p,p' are projections of a coplanar configuration 
of points 7r in space and E is the corresponding homog- 
raphy matrix Ep = p' , then s' T Ep = for all lines s' 
coincident with p' . 

Likewise, let W be the homography matrix due to 
7r, i.e., Wp = p" , then s" T Wp = for all lines s" co- 
incident with p" . Then, given the slicing of the ten- 
sor into three matrices (T t lk , T? k , T 3k ), then the tensors 
(W, 0, 0), (0, W, 0) and (0, 0, W) (and their linear combi- 
nations) all satisfy the constraint p's'^s'^T/ = 0. 

Therefore, the rank of the null space of the linear sys- 
tem of equations for the tensor is at least 6, since we have 
just created 6 'ghost' tensors. The question of whether 
the ghost tensors include the true tensor (in which case 
the rank of the estimation matrix is 21) or not (rank is 
20) is settled below. 

Theorem 3 The rank of the estimation matrix of the 
tensor from image measurements of three views of a pla- 
nar configuration of points is at most 21. 

proof: Denote the planar object by 7r and let E, W 
be the homography matrices due to 7r from view to 1, 
and from view to 2, respectively. The 'ghost' tensors 
due to E span the tensors of the form: 



T; 



jk 



8 k ei 



where 6 is a free vector of the family. Then, 

p i 8 ' j8 'tf k = (6 k 4)(p i 8 ' j ei) = 0. 

Likewise, The 'ghost' tensors due to W span the tensors 
of the form: 

where /j, is a free vector of the family. Then, 

The 6 dimensional null space spanned by both families 
of 'ghost' tensors spans the tensors of the form: 

<-jk 



T/ k =^w k +8 k ei 



where 6, [i are free vectors of the family. This family 
includes the true tensor (set /i = v' and 6 = v"). Thus, 
the rank of the estimation matrix is at most 27-6 = 21.Q 
Note, that unlike the case of LLC in which the ad- 
missibility constraints have reduced the ambiguity to a 
single solution, here the null space includes admissible 
tensors (admissibility constraints are satisfied), thus a 
unique solution is not possible. The ambiguity is also 
evident by straightforward counting: the tensor is deter- 
mined by 18 (algebraically independent) parameters, yet 
two homography matrices (E, W) give rise only to 8 pa- 
rameters each (because each matrix is up to scale), thus 
we have 2 parameters missing for uniquely determining 
the tensor from a planar surface. 

4 Experiments 

4.1 The experimental procedure 

Fig. 3a, 3b, and 3c show the three input images used for 
the experiments. The scene is composed of two faces of 
a cube and another plane on the left which is parallel to 
the vertical edge of the cube. This is a schematic model 
of a typical urban scene with an edge of a building such 
as in Fig la. 

Corresponding point features were manually ex- 
tracted. The feature points were saddle points formed by 
the corners of two black squares which can be found with 
subpixel accuracy. The point features were grouped into 
four groups: Points from the left and right faces of the 
cube form one group each. Points on the planar surface 
were grouped into two vertical sets of features. 

Line features were created by taking pairs of points. 
If no pair of points has members from more than one 
group (for example Fig. 3) then we limit ourselves to a 
Linear Line Complex since all the 3D lines in the scene 
intersect the edge of the cube. By adding pairs that 
span two groups we can add lines that do not belong to 
the LLC. By judiciously choosing pairs we can add lines 
that are close or far from being part of the LLC (see Fig. 
3d). We can also choose pairs of points that define lines 
passing through the epipoles. This flexibility allows us 
to verify all the claims in theorem 1. 

4.1.1 Hardware notes 

The images were captured using a Pulnix TM9701 
progressive scan camera with a 2/3 8nc ft CCD and an 
8-5 mm lens. The image resolution was 640 x 480 P j' :ce / s . 

To achieve the results presented here we had to take 
into account radial lens distortion. Only the first term of 
radial distortion was used. The radial distortion param- 
eter, K\ = 6e — 7 was found using the method described 
in [19]. We note that that parameter value also mini- 
mized the error terms in equation 3. 

4.2 Determining the LLC 

The three input images (Figures 3a, 3b, and 3c) will be 
denoted image 1, 2 and 3 respectively. We chose N = 28 
pairs of points which defined lines all belonging to the 
LLC. These are overlaid as white lines in the figures. 
For each image pair (1,2), (2,3) and (1,3) we used the 8 
point algorithm [7] applied to the line correspondences to 




Figure 3: The three input images used. All the lines marked are part of a linear line complex. They all intersect the 
line defined by the edge of the cube. Vertical lines intersect the edge at the point at infinity. The LLC was computed 
using images (b) and (c). The dashed lines in (b) and (c) are the projection of the common intersection line into the 
images. The results show it aligns very well with the edge of the true cube, (d) Three line which are not part of the 
LLC that are used in the experiments. 



Table 1: Values of the error cost function for estimating 
the LLC when all the lines belong to the LLC (none) and 
when we add a line which passes close to or far from the 
common line of intersection (the edge of the cube). 



Extra Line 


E23 


E12 


E13 


None 


0.000055 


0.00064 


0.000079 


Close 


0.0019 


0.0016 


0.00054 


Middle 


0.0065 


0.0038 


0.0017 


Far 


0.0119 


0.0919 


0.0062 



compute the matrices B12 , B23 and Bis that minimize: 

1 N 

Ei2=^j2( S > B ^y ( 3 ) 



^3 = ^:>> 5 ^;') 2 

8 = 1 

1 N 

Ei3=^j2( S > B ^ s "y 2 



respectively. The coordinates of the lines s,s',s" have 
been scaled as described in [7]. From theorem 1, the 
left and right null spaces of B23 (for example) are the 
projections of the line L in images 2 and 3. The dashed 
black line in figures (3b), and (3c) show the lines corre- 
sponding to the null spaces overlaid on the input images. 
They align well with the edge of the cube verifying the 
theory and showing that the matrix B can be recovered 
accurately. Similar results were found using the other 
image pairs. 

Fig. 3d shows image 1 on which we have overlaid three 
lines not belonging to the LLC. Table 1 shows the error 
terms of equations 3 when all the line are from the LLC 
and when we add one of the lines shown in figure 3d. 
When the extra line is far away from the common line of 
intersection the error is large. Even when the line nearly 
intersects the edge of the cube the error is still signifi- 
cant. Therefore if most of the line come from an LLC 
robust methods can be used to detect outliers. Other ex- 
periments, not reported here, use lines that pass through 
the epipole to verify the second half of theorem 1. 

4.3 Recovering motion and structure 

We computed the motion tensor from the three views 
using four methods. First we used the linear method 
for a set of 131 point correspondences. Then we used 
the linear method for a set of 34 non degenerate line 
correspondences. Next we applied the linear method in 
a naive way to the set-of 28 line correspondences from an 
LLC. In other words we ignored the fact that the lines 
come from an LLC. Finally we estimated the tensor from 
the 28 degenerate lines using the algorithm described in 
section 2.1. 

4.3.1 Condition of the estimation matrix 

Figure 4a (top) shows the four smallest singular values 
of the estimation matrix W used to compute the tensor 
from 34 non-degenerate lines. The smallest eigenvalue is 



considerably smaller than the others indicating that the 
null space of the matrix is of rank = 1 and the problem is 
well conditioned. Figure 4a (middle) shows the 5 small- 
est singular values for the estimation matrix computed 
from 28 lines belonging to an LLC. The 4 smallest singu- 
lar values are about equal and are considerably smaller 
than the next smallest. This indicates that the null space 
of W is of rank = 4 as expected from theorem 1. Sim- 
ply taking the eigenvector corresponding to the smallest 
eigenvalue would be a mistake. 

Figure 4b shows the projection of the vectors v\, i>2 
and i>3 on the eigenvectors of the estimation matrix 
W T W. The projections onto the eigenvectors corre- 
sponding to the first 23 eigenvalues are close to zero ver- 
ifying that the vectors v\, i>2 and i>3 are orthogonal to 
the first 23 eigenvectors. This verifies part of theorem 
2 which states that the vectors v\ , i>2 and i>3 are in the 
null space of W T W (the remaining four eigenvectors). 

Following the algorithm described in section 2.1 we 
compute the eigenvalues and eigenvectors of the matrix 



W T W 



«i«i 



v 2 vj 



v 3 vj . 



Figure 4a (bottom) shows the three smallest eigenvalues. 
The smallest eigenvalues is significantly smaller than the 
next smallest value indicating that the null space is now 
of rank = 1. 

4.3.2 Reprojection of lines using the tensor 

After recovering the tensor one can use the tensor 
to reproject a line given in two images into the third 
image. In order to test the tensor estimates we used ten 
additional lines shown in figure 5. Three of the lines lie 
in the LLC on the left face of the cube. The other 7 do 
not lie on the LLC. 

Figures 5b,c,d show the reprojection results (dashed 
lines) together with the original lines (solid lines) over- 
laid on image 1. One can see that if the set of lines used 
to estimate the tensor all belong to an LLC then other 
lines in the LLC reproject more or less correctly but the 
reprojection of lines not in the LLC is incorrect (fig. 5c). 
On the other hand, reprojection using the tensor com- 
puted by taking into account the degeneracy (fig. 5d) 
gives results as good as if we had a non-degenerate set 
of lines to estimate the tensor (fig. 5b). 

5 Summary 

We have shown that linear methods for estimating mo- 
tion and 3D structure from lines lead to a degenerate 
set of equations in the case of a Linear Line Complex. 
The LLC, a configuration of lines that all intersect a 
common line in V 3 , is in fact a common configuration 
of lines occuring frequently in man-made environments. 
This degeneracy is due to a bilinear constraint on lines 
in two views where the constraint equation has a form 
similar to the epipolar constraint but where lines replace 
points and the epipoles are replaced by the image of the 
common line of intersection in the two views. This con- 
straint can be used to determine whether a set of lines 
belongs to an LLC and enables us to reject a outliers 
using least median of squares or other robust estimation 
methods. 



Smallest eigenvalues {or 34 nondegenerate lines 




* $ 



10 15 

(b) 



Figure 4: (a) The smallest singular values of the estimation matrix W for a degenerate and non degenerate set of 
lines, (b) With a degenerate set of lines, the vectors vi, v% and v^, were projected onto the 21 eigenvectors ofW T W. 
The values for the first 23 eigenvectors are close to zero verifying that the vectors v\, i>2 and v% are orthogonal to the 
first 23 eigenvectors and are therefore in the null space ofW T W. (See theorem 2.) 



An LLC is not degenerate for non-linear methods in 
general. The theoretical analysis leads to a modifica- 
tion of the linear methods that can recover the structure 
and motion in the LLC case. We have proven that the 
motion can be recovered up to 8 discrete solutions. Em- 
pirical evidence shows that the number of solutions can 
be reduced further to a single solution one. 

We have implemented the algorithm, and experiments 
with real images verify the theoretical analysis. Al- 
though the results using the modified linear algorithm 
compare favorably with the results obtained using a non- 
degenerate set of lines, the system at this point is not 
robust. For example, it requires that lens distortion be 
taken into account. Further engineering would be in- 
volved in making a practical system. 

Acknowledgments 

A.S. wishes to thank Steve Maybank, Shai Avidan and 
Nassir Navab for helpful comments and acknowledge the 
financial support from US-IS Binational Science Foun- 
dation 94-00120/2, the European ACTS project AC074 
"Vanguard" , and from DARPA through ARL Contract 
DAAL01-97-0101. G.S. would like acknowledge the fi- 
nancial support from ONR contracts N00014-94-1-0128 
and DARPA contracts N00014-94-01-0994, N00014-97- 
0363. 

References 

[1] S. Avidan and A. Shashua. Tensorial transfer: On the 
representation of n > 3 views of a 3D scene. In Pro- 
ceedings of the ARPA Image Understanding Workshop, 
Palm Springs, CA, February 1996. 

S. Avidan and A. Shashua. View synthesis in tensor 
space. In Proceedings of the IEEE Conference on Com- 



m 



puter Vision and Pattern Recognition, Puerto Rico, June 
1997. 

[3] T. Buchanan. On the critical set for photogrammetric 
reconstruction using line tokens in p3(c). Geometriae 
Dedicata, 44:223-232, 1992. 

[4] O.D. Faugeras and B. Mourrain. On the geometry and 
algebra of the point and line correspondences between N 
images. In Proceedings of the International Conference 
on Computer Vision, Cambridge, MA, June 1995. 

[5] O.D. Faugeras and T. Papadopoulo. A nonlinear method 
for estimating the projective geometry of three views. To 
be published ICCV 98, January 1998. 

[6] R. Hartley. Lines and points in three views — a unified 
approach. In Proceedings of the ARPA Image Under- 
standing Workshop, Monterey, CA, November 1994. 

[7] R. Hartley. A linear method for reconstruction from lines 
and points. In Proceedings of the International Confer- 
ence on Computer Vision, pages 882-887, Cambridge, 
MA, June 1995. 

[8] A. Heyden. Reconstruction from image sequences by 
means of relative depths. In Proceedings of the Inter- 
national Conference on Computer Vision, pages 1058- 
1063, Cambridge, MA, June 1995. 

[9] S.J. Maybank. The critical line concruence for re- 
construction from three images. Applicable Algebra in 
Engineering Communication and Computing (AAECC), 
6:89-113, 1995. 

[10] N. Navab, O.D. Faugeras, and T. Vieville. The criti- 
cal sets of lines for camera displacement estimation: A 
mixed Euclidean-projective and constructive approach. 
In Proceedings of the International Conference on Com- 
puter Vision, Berlin, Germany, May 1993. 

[11] A. Shashua. Algebraic functions for recognition. IEEE 
Transactions on Pattern Analysis and Machine Intelli- 
gence, 17(8):779-789, 1995. 




Figure 5: (a) Ten additional lines are used to test the recovered tensors. Three of the lines lie on the left face of 
the cube and are therefore the part of the LLC. The other 7 are not. The test lines from images (2) and (3) are 
reprojected back into image (1) using the recovered tensors. Solid lines are the true locations. Dashed lines are the 
reprojected lines, (b) The tensor was computed using a set of 34 non degenerate lines, (c) The tensor was computed 
using a degenerate set of 28 lines using the standard method. Since it does not take into account the degeneracy the 
tensor fails to correctly reproject the lines. The only lines in (c) that reproject correctly are those that belong to the 
LLC. (d) Using the new algorithm for the degenerate case the computed tensor correctly reprojects all the lines. 



[12] A. Shashua. Trilinear tensor: The fundamental con- 
struct of multiple-view geometry and its applications. 
Submitted for journal publication, 1997. 

[13] A. Shashua and P. Anandan. The generalized trilin- 
ear constraints and the uncertainty tensor. In Pro- 
ceedings of the ARPA Image Understanding Workshop, 
Palm Springs, CA, February 1996. 

[14] A. Shashua and S. Avidan. The rank4 constraint in 
multiple view geometry. In Proceedings of the European 
Conference on Computer Vision, Cambridge, UK, April 
1996. 

[15] A. Shashua and M. Werman. Trilinearity of three per- 
spective views and its associated tensor. In Proceed- 
ings of the International Conference on Computer Vi- 
sion, June 1995. 

[16] M.E. Spetsakis and J. Aloimonos. Structure from mo- 
tion using line correspondences. International lournal 
of Computer Vision, 4(3):171-183, 1990. 

[17] M.E. Spetsakis and J. Aloimonos. A unified theory of 
structure from motion. In Proceedings of the ARPA Im- 
age Understanding Workshop, 1990. 

[18] G. Stein and A. Shashua. Model based brightness con- 
straints: On direct estimation of structure and motion. 
In Proceedings of the IEEE Conference on Computer Vi- 
sion and Pattern Recognition, Puerto Rico, June 1997. 

[19] G. Stein. Lens distortion calibration using point corre- 
spondences. In Proceedings of the IEEE Conference on 
Computer Vision and Pattern Recognition, Puerto Rico, 
June 1997. 

[20] B. Triggs. Matching constraints and the joint image. 
In Proceedings of the International Conference on Com- 
puter Vision, pages 338-343, Cambridge, MA, June 
1995. 

[21] J. Weng, T.S. Huang, and N. Ahuja. Motion and struc- 
ture from line correspondences: Closed form solution, 
uniqueness and optimization. IEEE Transactions on 
Pattern Analysis and Machine Intelligence, 14(3), 1992. 

A Mathematical Background and the 
Trilinear Tensor 

Let a; be a point in 3D space and its projection in a pair 
of images be p and p' . Then p = [I; 0]x and p' = Ax, 
where = denotes equality up to scale. The left 3x3 
minor of A stands for a 2D projective transformation of 
the chosen plane at infinity and the fourth column of 
A stands for the epipole (the projection of the center of 
camera on the image plane of camera 1). In particular, 
in a calibrated setting the 2D projective transformation 
is the rotational component of camera motion and the 
epipole is the translational component of camera motion. 
We will occasionally use tensorial notations as de- 
scribed next. We use the covariant-contravariant sum- 
mation convention: a point is an object whose co- 
ordinates are specified with superscripts, i.e., p % = 
{p l ,p 2 , ■■■)■ These are called contravariant vectors. An 
element in the dual space (representing hyperplanes — 
lines in V 2 ), is called a covariant vector and is repre- 
sented by subscripts, i.e., Sj = (si, S2, ....). Indices re- 
peated in covariant and contravariant forms are summed 
over, i.e., p % S( = p 1 s\ +p 2 S'j + ...+p"s„. This is known as 
a contraction. An outer-product of two 1-valence tensors 



(vectors), a,{V , is a 2- valence tensor (matrix) c^ whose 
i, j entries are a 8 & J — note that in matrix form C = ba T . 

Matching image points across three views will be 
denoted by p,p',p"; the homogeneous coordinates will 
be referred to as p l ,p' J ,p" k , or alternatively as non- 
homogeneous image coordinates (x, y), (x' , y'), (x" , y") 
— hence, p % = (x, y, 1), etc. 

Three views, p = [l;0]x,p' = Ax and p" = Bx, are 
known to produce four trilinear forms whose coefficients 
are arranged in a tensor representing a bilinear function 
of the camera matrices A, B: 

' jk - v' j b k 
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where A = [a{,v' j ] (a{ is the 3 x 3 left 

the fourth column of A) and B = [bf 

acts on a triplet of matching points in the following way: 
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where s^ are any two lines {s l - and s 2 ) intersecting at 
p' , and r p k are any two lines intersecting p" . Since the 
free indices are /i,p each in the range 1,2, we have 4 
trilinear equations (unique up to linear combinations). If 
we choose the standard form where s M (and r p ) represent 
vertical and horizontal scan lines, i.e., 

-1 x' 

-1 y' 

then the four trilinear forms, referred to as trilmearities 
[11], have the following explicit form: 

V7; 3 y + x't? 1 ^ 

13 - 8 ' j/V7; 3 y + x'Tpj 

y"T 23 P t -y"y'Tr P t + y'Tr P t 

These constraints were first derived in [11]; the tensorial 
derivation leading to eqns. 4 and 5 was first derived in 
[13]. The trilinear tensor has been well known in dis- 
guise in the context of Euclidean line correspondences 
and was not identified at the time as a tensor but as a 
collection of three matrices (a particular contraction of 
the tensor, correlation contractions, as explained next) 
[16, 17, 21]. The link between the two and the general- 
ization to projective space was identified later by Hart- 
ley [6, 7]. Additional work in this area can be found in 
[15, 4, 20, 8, 14, 1], and applications in [2, 18]. 

The tensor has certain contraction properties and can 
be sliced in three principled ways into matrices with 
distinct geometric properties. These properties is what 
makes the tensor distinct from simply being a collection 
of three matrices and will be briefly discussed next — 
further details can be found in [12]. 

A.l Contraction Properties and Tensor Slices 

Consider the matrix arising from the contraction, 

W* (6) 

which is a 3 x 3 matrix, we denote by E, obtained by the 
linear combination E = 6iT? + 82T/ + 63T? (which is 
what is meant by a contraction), and 8k is an arbitrary 
covariant vector. The matrix E has a general meaning 
introduced in [15]: 



Proposition 1 (Homography Contractions) The 

contraction 8kT/ for some arbitrary 8k is a homogra- 
phy matrix from image one onto image two determined 
by the plane containing the third camera center C" and 
the line 8k in the third image plane. Generally, the rank 
of E is 3. Likewise, the contraction 8jT? is a homog- 
raphy matrix from image one onto image three. 

For proof see [15]. Clearly, since 8 is spanned by three 
vectors, we can generate up to at most three distinct 
homography matrices by contractions of the tensor. We 
define the Standard Homography Slicing as the homog- 
raphy contractions associated by selecting 8 be (1,0,0) 
or (0, 1, 0) or (0, 0, 1), thus the three standard homogra- 
phy slices between image one and two are T? , T? and 
T/ , and we denote them by E\, E^, £3 respectively, and 
likewise the three standard homography slices between 
image one and three are T t lk ,T? k and T t 3k , and we de- 
note them by W\, W2, W3 respectively. 
Similarly, consider the contraction 

6'T> k (7) 

which is a 3 x 3 matrix, we denote by T, and where 8 l 
is an arbitrary contravariant vector. The matrix T has 
a general meaning is well, as detailed below [12]: 

Proposition 2 The contraction 8'T/ for some arbi- 
trary 8' is a rank 2 correlation matrix from image two 
onto image three, that maps the dual image plane (the 
space of lines in image two) onto a set of collm ear points 
in image three that form the epipolar line corresponding 
to the point 8 l in image one. The null space of the cor- 
relation matrix is the epipolar line of 8 l in image two. 
Similarly, the transpose of T is a correlation from image 
three onto image two with the null space being the epipo- 
lar line in image three corresponding to the point 8 l in 
image one. 

For proof see [12]. We define the Standard Correlation 
Slicing as the correlation contractions associated with se- 
lecting 8 be (1, 0, 0) or (0, 1, 0) or (0, 0, 1), thus the three 
standard correlation slices are T( , T| and T£ , and we 
denote them by Ti,T2,Ts, respectively. The three stan- 
dard correlations date back to the work on structure 
from motion of lines across three views [16, 21] where 
these matrices were first introduced. 

A. 2 Tensor Admissibility Constraints 

The 27 coefficients T/ are not independent. One can 
easily show that the tensor is determined by only 18 pa- 
rameters; and from the contraction properties discussed 
above that the constraints among the 27 coefficients, re- 
ferred to as admissibility constraints, are grouped into 
three classes. Both will be discussed briefly below (fur- 
ther details in [12]). 

A. 2.1 18 Parameters 

The tensor 



parameters drop out because we can scale v' and ac- 
cordingly bf without changing the tensor, and likewise 
scale v" and accordingly a\ . An additional parameter 
drops out because of the global scale factor (tensor is 
determined up to overall scale). Thus, we readily see 
there can be at most 21 parameters defining the tensor. 
We can drop out three more parameters by noticing that 
the matrices a\ and bf belong to a family of homography 
matrices that leaves the tensor unchanged (uniqueness 
proof in [11]), as detailed below: 



%■ 



jk 



v"bf-v" k a\ (8) 

v'^bf +a i v" k )-v" k (ai+a i v'i) (9) 
T/ k + aiV ,j v" k -aiv' j v" k (10) 
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hence, we have three free parameters a.{ (in geometric 
terms there is a free choice of reference plane in space). 
We can select a 8 - such that the matrix bf will have a 
vanishing column (this corresponds to selecting a refer- 
ence plane coplanar with the center of projection of the 
third view). Therefore, the new matrices a\ and bf have 
only 15 non- vanishing entries, and we have reduced the 
number of parameters from 21 to 18. 

A. 2. 2 Admissibility Constraints 

We may deduce from the Correlation Contractions 
discussed above the following three groups of constraints 
that the 27 coefficients must satisfy: 

1. Rank(<5 8 7^ )=2 for all choices of 8. The three stan- 
dard correlation slices T\ , T 2 , T3 are of rank 2 each 
and this property is closed under all linear combi- 
nations. 

2. Rank(null(Ti),null(T 2 ),null(T 3 ))=2. This follows 
from the fact the the null space of 8'T/ is the 
epipolar line in the second image corresponding to 
the point 8 in the first image — since all epipolar 
lines are concurrent, their rank is 2. 

3. Rank(null(T 1 T ),null(T 2 T ),null(T 3 T ))=2. These are 
epipolar lines in third image, thus their rank is 2 
as well. 

One can easily show that no subset of these con- 
straints is sufficient to describe an admissible tensor of 
the form of eqn. 4. In practice, in the presence of er- 
rors in image measurements one often starts with the 
Linear solution (that might not satisfy the admissibil- 
ity constraints) and improves it further by employing a 
numerical Gauss-Newton style iterative procedure until 
a solution that satisfies the admissibility constraints is 
obtained (for example, [5]). 



T: 



jk 



v ,j b k 



v" k d 



is determined by 24 parameters given by the two cam- 
era matrices, each has 12 parameters. Two additional 
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